Artificial intelligence enhanced system for adaptive control driven ar/vr visual aids

ABSTRACT

Interactive systems using adaptive control software and hardware from known and later developed eyepieces to later developed head-wear to lenses, including implantable, temporarily insertable and contact and related film based types of lenses including thin film transparent elements for housing cameras lenses and projector and functional equivalent processing tools. Simple controls, real-time updates and instant feedback allow implicit optimization of a universal model while managing complexity.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/727,564, filed Dec. 26, 2019, now U.S. Pat. No. 11,043,036, which isa continuation of U.S. patent application Ser. No. 16/030,788, filedJul. 9, 2018, now abandoned, which application claims the benefit of andpriority to U.S. Provisional Patent Application Nos. 62/530,286 and62/530,792, filed July, 2017, the content of each of which isincorporated herein by reference in its entirely.

BACKGROUND OF THE DISCLOSURES

The Interactive Augmented Reality (AR) Visual Aid invention describedbelow is intended for users with visual impairments that impact field ofvision (FOV). These may take the form of age-related maculardegeneration, retinitis pigmentosa, diabetic retinopathy, Stargardt'sdisease, and other diseases where damage to part of the retina impairsvision. The invention described is novel because it not only suppliesalgorithms to enhance vision, but also provides simple but powerfulcontrols and a structured process that allows the user to adjust thosealgorithms.

OBJECTS AND SUMMARY OF THE INVENTION

The basic hardware is constructed from a non-invasive, wearableelectronics-based AR eyeglass system (see FIG. 2) employing any of avariety of integrated display technologies, including LCD, OLED, ordirect retinal projection. One or more cameras, mounted on the glasses,continuously monitor the view where the glasses are pointing. The ARsystem also contains an integrated processor and memory storage (eitherembedded in the glasses, or tethered by a cable) with embedded softwareimplementing real-time algorithms that modify the images as they arecaptured by the camera(s). These modified, or corrected, images are thencontinuously presented to the eyes of the user via the integrateddisplays.

The basic image modification algorithms come in multiple forms asdescribed later. In conjunction with the AR hardware glasses, theyenable users to enhance vision in ways extending far beyond simple imagechanges such as magnification or contrast enhancement. The fundamentalinvention is a series of adjustments that are applied to move, modify,or reshape the image in order to reconstruct it to suit each specificuser's FOV and take full advantage of the remaining useful retinal area.The following disclosure describes a variety of mapping, warping,distorting and scaling functions used to correct the image for the enduser.

The invention places these fundamental algorithms under human control,allowing the user to interact directly with the corrected image andtailor its appearance for their particular condition or specific usecase. In prior art, an accurate map of the usable user FOV is a requiredstarting point that must be known in order to provide a template formodifying the visible image. With this disclosure, such a detailedstarting point derived from FOV measurements does not have to besupplied. Instead, an internal model of the FOV is developed, beginningwith the display of a generic template or a shape that is believed toroughly match the type of visual impairment of the user. From thissimple starting point the user adjusts the shape and size of thedisplayed visual abnormality, using the simple control interface to adddetail progressively, until the user can visually confirm that thedisplayed model captures the nuances of his or her personal visualfield. Using this unique method, accurate FOV tests and initialtemplates are not required. Furthermore, the structured process, whichincrementally increases model detail, makes the choice of initial modelnon-critical.

For people with retinal diseases, adapting to loss a vision becomes away of life. This impact can affect their life in many ways includingloss of the ability to read, loss of income, loss of mobility and anoverall degraded quality of life. However, with prevalent retinaldiseases such as AMD (Age related Macular Degeneration) not all of thevision is lost, and in this case the peripheral vision remains intact asonly the central vision is impacted with the degradation of the macula.Given that the peripheral vision remains intact it is possible to takeadvantage of eccentric viewing and through patient adaptation toincrease functionality such as reading. Research has proven that throughtraining of the eccentric viewing increased reading ability (bothaccuracy and speed). Eye movement control training and PRL (PreferredRetinal Locus) training were important to achieving these results.Another factor in increasing reading ability with those with reducedvision is the ability to views words in context as opposed to isolation.Magnification is often used as a simply visual aid with some success.However, with increased magnification comes decreased FOV (Field ofView) and therefore the lack of ability to see other words or objectsaround the word or object of interest. Although it was proven that withextensive training isolated word reading can improve, eye control wasimportant to this as well. The capability to guide the training foreccentric viewing and eye movement and fixation training is important toachieve the improvement in functionality such as reading. Theseapproaches outlined below will serve to both describe novel ways to useaugmented reality techniques to both automate and improve the training.

In order to help users with retinal diseases, especially users withcentral vision deficiencies. First it is important to train and helptheir ability to fixate on a target. Since central vision is normallyused for this, this is an important step to help users control theirability to focus on a target. Thereby laying the ground work for moretraining and adaptation functionality. This fixation training can beaccomplished through gamification built into the software algorithms,and can be utilized periodically for increased fixation training andimproved adaptation. The gamification can be accomplished by followingfixation targets around the display screen and in conjunction with ahand held pointer can select or click on the target during timed oruntimed exercise. Furthermore, this can be accomplished through voiceactive controls as a substitute or adjunct to a hand help pointer.

To aid the user in targeting and fixation certain guide lines can beoverlaid on reality or on the incoming image to help guide the users eyemovements along the optimal path. These guidelines can be a plurality ofconstructs such as, but not limited to, cross hair targets, bullseyetargets or linear guidelines such as singular or parallel dotted linesof a fixed or variable distance apart, a dotted line or solid box ofvarying colors. This will enable the user to increase their training andadaptation for eye movement control to following the tracking lines ortargets as their eyes move across a scene in the case of a landscape,picture or video monitor or across a page in the case of reading text.

This approach can be further modified and improved with otherinteractive methods beyond simple eye movement. Targeting approaches asdescribed above can also be tied to head movement based on inertialsensor inputs or simply following along as the head moves.

Furthermore, these guided fixation targets, or lines, can move acrossthe screen at a predetermined fixed rate to encourage the user to followalong and keep pace. These same targets can also be scrolled across thescreen at variable rates as determined or triggered by the user forcustomization to the situation or scene or text of interest.

To make the most of a user's remaining useful vision methods foradaptive peripheral vision training can be employed. Training andencouraging the user to make the most of their eccentric viewingcapabilities is important. As described the user may naturally gravitateto their PRL (preferred retinal locus) to help optimized their eccentricviewing. However, this may not be the optimal location to maximize theirability to view images or text with their peripheral vision. Through useof skewing and warping the images presented to the user, along with thetargeting guidelines it can be determined where the optimal place forthe user to target their eccentric vision.

Eccentric viewing training through reinforced learning can be encouragedby a series of exercises. The targeting as described in fixationtraining can also be used for this training. With fixation targets onand the object, area, or word of interest can be incrementally tested byshifting locations to determine the best PRL for eccentric viewing.

Also, pupil tracking algorithms can be employed and not only have eyetracking capability but can also utilize user customized offset forimproved eccentric viewing capability. Whereby the eccentric viewingtargets are offset guide the user to focus on their optimal area foreccentric viewing.

Further improvements in visual adaptation can be achieved through use ofthe hybrid distortion algorithms. With the layered distortion approachobjects or words on the outskirts of the image can receive a differentdistortion and provide a look ahead preview to piece together words forincreased reading speed. While the user is focused on the area ofinterest that is being manipulated the words that are moving into thefocus area can help to provide context in order to interpolate andbetter understand what is coming for faster comprehension and contextualunderstanding.

Furthermore, the user can be run through a series of practice moduleswhereby different distortion levels and methods are employed. With thesedifferent methods hybrid distortion training can be used to switchbetween areas of interest to improve fixation.

BRIEF DESCRIPTION OF THE DRAWINGS

Various preferred embodiments are described herein with references tothe drawings in which merely illustrative views are offered forconsideration, whereby:

FIG. 1 is a grid manipulation flowchart, including hierarchical modelconstruction.

FIG. 2 is a flowchart of an eyeglass system that continuously presentsmodified images to the eyes of a user.

FIGS. 3A-3B depict front and side views of one eye within a set ofnotional OST AR glasses employing a half-silvered mirror to combineincoming scene light with the image shown on an internally-mounteddisplay.

Corresponding reference characters indicate corresponding components arenot needed throughout the single view of the drawing. Skilled artisanswill appreciate that elements in the figures are illustrated forsimplicity and clarity, and have not necessarily been drawn to scale.For example, the dimensions of some of the elements in the figures maybe exaggerated relative to other elements to help to improveunderstanding of various embodiments of the present invention. Also,common but well-understood elements that are useful or necessary in acommercially feasible embodiment are often not depicted in order tofacilitate a less obstructed view of these various embodiments of thepresent invention.

DETAILED DESCRIPTIONS

The present inventors have discovered that low-vision users can conforma user-tuned software set and improve needed aspects of vision to enablefunctional vision to be restored.

Expressly incorporated by reference as if fully set forth herein are thefollowing: U.S. Provisional Patent Application No. 62/530,286 filed Jul.9, 2017, U.S. Provisional Patent Application No. 62/530,792 filed Jul.9, 2017, U.S. Provisional Patent Application No. 62/579,657, filed Oct.13, 2017, U.S. Provisional Patent Application No. 62/579,798, filed Oct.13, 2017, Patent Cooperation Treaty Patent Application No.PCT/US2017/062421, filed Nov. 17, 2017, U.S. patent application Ser. No.15/817,117, filed Nov. 17, 2017, U.S. Provisional Patent Application No.62/639,347, filed Mar. 6, 2018, U.S. patent application Ser. No.15/918,884, filed Mar. 12, 2018, and U.S. Provisional Patent ApplicationNo. 62/677,463, filed May 29, 2018.

It is contemplated that the processes described above are implemented ina system configured to present an image to the user. The processes maybe implemented in software, such as machine readable code or machineexecutable code that is stored on a memory and executed by a processor.Input signals or data is received by the unit from a user, cameras,detectors or any other device. Output is presented to the user in anymanner, including a screen display or headset display

In addition, the logic flows depicted in the figures do not require theparticular order shown, or sequential order, to achieve desirableresults. Furthermore, other steps may be provided or steps may beeliminated from the described flows, and other components may be addedto, or removed from, the described systems. Accordingly, otherembodiments are within the scope of the following claims.

Referring now to FIG. 1, systems of the present invention are shownschematically. Steps A-I are enhanced by the various interfaces andloops connecting AI interfaces with the instant system, as is known tothose of skill in the art.

AI Data 101, resides both in its own database and in an AI cloud 109,along with AI Compiler 111, and AI filter 107 along with any otherrequired AI architecture 103 and AI Intervenor 105. Step A involvesidentifying region(s) to remap from with source FOV; Step B initializingthe same to achieve Step C wherein the model created is ratified.

AI Architecture 103 provides both resident and transient data sets toaddress the issue(s) being ameliorated in the user's vision. Said datasets reside in at least one of the sub-elements of the AI architecture,namely AI cloud 109, AI compiler 111, AI filter 107 and AI intervenor105, as known to those skilled in the art. Likewise, Step D wherein userselects point outputs, and step E wherein user moves selected point(s)updating models in real-time, and Step F, wherein user releases selectedpoint(s), along with step G wherein interlocutory model is deemedcomplete, or H needing updates or I complete. Those skilled in the artunderstand the multi-path approach and orientation to use AI elements tocreate functional and important models using said data, inter alia.

It will be appreciated that the above embodiments that have beendescribed in particular detail are merely example or possibleembodiments, and that there are many other combinations, additions, oralternatives that may be included. For example, while online gaming hasbeen referred to throughout, other applications of the above embodimentsinclude online or web-based applications or other cloud services.

Also, the particular naming of the components, capitalization of terms,the attributes, data structures, or any other programming or structuralaspect is not mandatory or significant, and the mechanisms thatimplement the invention or its features may have different names,formats, or protocols. Further, the system may be implemented via acombination of hardware and software, as described, or entirely inhardware elements. Also, the particular division of functionalitybetween the various system components described herein is merelyexemplary, and not mandatory; functions performed by a single systemcomponent may instead be performed by multiple components, and functionsperformed by multiple components may instead be performed by a singlecomponent.

Some portions of the above description present features in terms ofalgorithms and symbolic representations of operations on information.These algorithmic descriptions and representations may be used by thoseskilled in the data processing arts to most effectively convey thesubstance of their work to others skilled in the art. These operations,while described functionally or logically, are understood to beimplemented by computer programs. Furthermore, it has also provenconvenient at times, to refer to these arrangements of operations asmodules or by functional names, without loss of generality.

Unless specifically stated otherwise as apparent from the abovediscussion, it is appreciated that throughout the description,discussions utilizing terms such as “processing” or “computing” orcalculating” or “determining” or “identifying” or “displaying” or“providing” or the like, refer to the action and processes of a computersystem, or similar electronic computing device, that manipulates andtransforms data represented as physical (electronic) quantities withinthe computer system memories or registers or other such informationstorage, transmission or display devices.

Based on the foregoing specification, the above-discussed embodiments ofthe invention may be implemented using computer programming orengineering techniques including computer software, firmware, hardwareor any combination or subset thereof. Any such resulting program, havingcomputer-readable and/or computer-executable instructions, may beembodied or provided within one or more computer-readable media, therebymaking a computer program product, i.e., an article of manufacture,according to the discussed embodiments of the invention. The computerreadable media may be, for instance, a fixed (hard) drive, diskette,optical disk, magnetic tape, semiconductor memory such as read-onlymemory (ROM) or flash memory, etc., or any transmitting/receiving mediumsuch as the Internet or other communication network or link. The articleof manufacture containing the computer code may be made and/or used byexecuting the instructions directly from one medium, by copying the codefrom one medium to another medium, or by transmitting the code over anetwork.

While the disclosure has been described in terms of various specificembodiments, it will be recognized that the disclosure can be practicedwith modification within the spirit and scope of the claims.

While several embodiments of the present disclosure have been describedand illustrated herein, those of ordinary skill in the art will readilyenvision a variety of other means and/or structures for performing thefunctions and/or obtaining the results and/or one or more of theadvantages described herein, and each of such variations and/ormodifications is deemed to be within the scope of the presentdisclosure. More generally, those skilled in the art will readilyappreciate that all parameters, dimensions, materials, andconfigurations described herein are meant to be exemplary and that theactual parameters, dimensions, materials, and/or configurations willdepend upon the specific application or applications for which theteachings of the present disclosure is/are used.

Those skilled in the art will recognize, or be able to ascertain usingno more than routine experimentation, many equivalents to the specificembodiments of the disclosure described herein. It is, therefore, to beunderstood that the foregoing embodiments are presented by way ofexample only and that, within the scope of the appended claims andequivalents thereto, the disclosure may be practiced otherwise than asspecifically described and claimed. The present disclosure is directedto each individual feature, system, article, material, kit, and/ormethod described herein. In addition, any combination of two or moresuch features, systems, articles, materials, kits, and/or methods, ifsuch features, systems, articles, materials, kits, and/or methods arenot mutually inconsistent, is included within the scope of the presentdisclosure.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.” The phrase“and/or,” as used herein in the specification and in the claims, shouldbe understood to mean “either or both” of the elements so conjoined,i.e., elements that are conjunctively present in some cases anddisjunctively present in other cases. Other elements may optionally bepresent other than the elements specifically identified by the “and/or”clause, whether related or unrelated to those elements specificallyidentified, unless clearly indicated to the contrary.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment. Thus, appearances of the phrases “in oneembodiment” or “in an embodiment” in various places throughout thisspecification are not necessarily all referring to the same embodiment.Furthermore, the particular features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments.

The terms and expressions which have been employed herein are used asterms of description and not of limitation, and there is no intention,in the use of such terms and expressions, of excluding any equivalentsof the features shown and described (or portions thereof), and it isrecognized that various modifications are possible within the scope ofthe claims. Accordingly, the claims are intended to cover all suchequivalents.

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present invention. Thus,appearances of the phrases “in one embodiment,” “in an embodiment,” andsimilar throughout this specification may, but do not necessarily, allrefer to the same embodiment.

Furthermore, the described features, structures, or characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. In the following description, numerous specific details areprovided to provide a thorough understanding of embodiments of theinvention. One skilled in the relevant art will recognize, however, thatthe invention may be practiced without one or more of the specificdetails, or with other methods, components, materials, and so forth. Inother instances, well-known structures, materials, and so forth. Inother instances, well-known structures, materials, or operations are notshown or described in detail to avoid obscuring aspects of theinvention.

The schematic flow chart diagrams included herein are generally setforth as logical flow chart diagrams. As such, the depicted order andlabeled steps are indicative of one embodiment of the presented method.Other steps and methods may be conceived that are equivalent infunction, logic, or effect to one or more steps, or portions thereof, ofthe illustrated method. Additionally, the format and symbols employedare provided to explain the logical steps of the method and areunderstood not to limit the scope of the method. Although various arrowtypes and line types may be employed in the flow chart diagrams, theyare understood not to limit the scope of the corresponding method.Indeed, some arrows or other connectors may be used to indicate only thelogical flow of the method. For instance, an arrow may indicate awaiting or monitoring period of unspecified duration between enumeratedsteps of the depicted method. Additionally, the order in which aparticular method occurs may or may not strictly adhere to the order ofthe corresponding steps shown. Unless otherwise indicated, all numbersexpressing quantities of ingredients, properties such as molecularweight, reaction conditions, and so forth used in the specification andclaims are to be understood as being modified in all instances by theterm “about.” Accordingly, unless indicated to the contrary, thenumerical parameters set forth in the specification and attached claimsare approximations that may vary depending upon the desired propertiessought to be obtained by the present invention. At the very least, andnot as an attempt to limit the application of the doctrine ofequivalents to the scope of the claims, each numerical parameter shouldat least be construed in light of the number of reported significantdigits and by applying ordinary rounding techniques. Notwithstandingthat the numerical ranges and parameters setting forth the broad scopeof the invention are approximations, the numerical values set forth inthe specific examples are reported as precisely as possible. Anynumerical value, however, inherently contains certain errors necessarilyresulting from the standard deviation found in their respective testingmeasurements.

The terms “a,” “an,” “the” and similar referents used in the context ofdescribing the invention (especially in the context of the followingclaims) are to be construed to cover both the singular and the plural,unless otherwise indicated herein or clearly contradicted by context.Recitation of ranges of values herein is merely intended to serve as ashorthand method of referring individually to each separate valuefalling within the range. Unless otherwise indicated herein, eachindividual value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g., “such as”) provided herein isintended merely to better illuminate the invention and does not pose alimitation on the scope of the invention otherwise claimed. No languagein the specification should be construed as indicating any non-claimedelement essential to the practice of the invention.

Groupings of alternative elements or embodiments of the inventiondisclosed herein are not to be construed as limitations. Each groupmember may be referred to and claimed individually or in any combinationwith other members of the group or other elements found herein. It isanticipated that one or more members of a group may be included in, ordeleted from, a group for reasons of convenience and/or patentability.When any such inclusion or deletion occurs, the specification is deemedto contain the group as modified thus fulfilling the written descriptionof all Markush groups used in the appended claims.

Certain embodiments of this invention are described herein, includingthe best mode known to the inventors for carrying out the invention. Ofcourse, variations on these described embodiments will become apparentto those of ordinary skill in the art upon reading the foregoingdescription. The inventor expects skilled artisans to employ suchvariations as appropriate, and the inventors intend for the invention tobe practiced otherwise than specifically described herein. Accordingly,this invention includes all modifications and equivalents of the subjectmatter recited in the claims appended hereto as permitted by applicablelaw. Moreover, any combination of the above-described elements in allpossible variations thereof is encompassed by the invention unlessotherwise indicated herein or otherwise clearly contradicted by context.

Specific embodiments disclosed herein may be further limited in theclaims using consisting of or consisting essentially of language. Whenused in the claims, whether as filed or added per amendment, thetransition term “consisting of” excludes any element, step, oringredient not specified in the claims. The transition term “consistingessentially of” limits the scope of a claim to the specified materialsor steps and those that do not materially affect the basic and novelcharacteristic(s). Embodiments of the invention so claimed areinherently or expressly described and enabled herein.

As one skilled in the art would recognize as necessary or best-suitedfor performance of the methods of the invention, a computer system ormachines of the invention include one or more processors (e.g., acentral processing unit (CPU) a graphics processing unit (GPU) or both),a main memory and a static memory, which communicate with each other viaa bus.

A processor may be provided by one or more processors including, forexample, one or more of a single core or multi-core processor (e.g., AMDPhenom II X2, Intel Core Duo, AMD Phenom II X4, Intel Core i5, IntelCore I & Extreme Edition 980X, or Intel Xeon E7-2820).

An I/O mechanism may include a video display unit (e.g., a liquidcrystal display (LCD) or a cathode ray tube (CRT)), an alphanumericinput device (e.g., a keyboard), a cursor control device (e.g., amouse), a disk drive unit, a signal generation device (e.g., a speaker),an accelerometer, a microphone, a cellular radio frequency antenna, anda network interface device (e.g., a network interface card (NIC), Wi-Ficard, cellular modem, data jack, Ethernet port, modem jack, HDMI port,mini-HDMI port, USB port), touchscreen (e.g., CRT, LCD, LED, AMOLED,Super AMOLED), pointing device, trackpad, light (e.g., LED), light/imageprojection device, or a combination thereof.

Memory according to the invention refers to a non-transitory memorywhich is provided by one or more tangible devices which preferablyinclude one or more machine-readable medium on which is stored one ormore sets of instructions (e.g., software) embodying any one or more ofthe methodologies or functions described herein. The software may alsoreside, completely or at least partially, within the main memory,processor, or both during execution thereof by a computer within system,the main memory and the processor also constituting machine-readablemedia. The software may further be transmitted or received over anetwork via the network interface device.

While the machine-readable medium can in an exemplary embodiment be asingle medium, the term “machine-readable medium” should be taken toinclude a single medium or multiple media (e.g., a centralized ordistributed database, and/or associated caches and servers) that storethe one or more sets of instructions. The term “machine-readable medium”shall also be taken to include any medium that is capable of storing,encoding or carrying a set of instructions for execution by the machineand that cause the machine to perform any one or more of themethodologies of the present invention. Memory may be, for example, oneor more of a hard disk drive, solid state drive (SSD), an optical disc,flash memory, zip disk, tape drive, “cloud” storage location, or acombination thereof. In certain embodiments, a device of the inventionincludes a tangible, non-transitory computer readable medium for memory.Exemplary devices for use as memory include semiconductor memorydevices, (e.g., EPROM, EEPROM, solid state drive (SSD), and flash memorydevices e.g., SD, micro SD, SDXC, SDIO, SDHC cards); magnetic disks,(e.g., internal hard disks or removable disks); and optical disks (e.g.,CD and DVD disks).

Furthermore, numerous references have been made to patents and printedpublications throughout this specification. Each of the above-citedreferences and printed publications are individually incorporated hereinby reference in their entirety.

In closing, it is to be understood that the embodiments of the inventiondisclosed herein are illustrative of the principles of the presentinvention. Other modifications that may be employed are within the scopeof the invention. Thus, by way of example, but not of limitation,alternative configurations of the present invention may be utilized inaccordance with the teachings herein. Accordingly, the present inventionis not limited to that precisely as shown and described.

Augmented Reality—VST vs. OST. Augmented Reality (AR) eyewearimplementations fall cleanly into two disjoint categories, videosee-through (VST) and optical see-through (OST).

Apparatus for VST AR closely resembles Virtual Reality (VR) gear, wherethe wearer's eyes are fully enclosed so that only content directly shownon the embedded display remains visible. VR systems maintain afully-synthetic three-dimensional environment that must be continuouslyupdated and rendered at tremendous computational cost. In contrast, VSTAR instead presents imagery based on the real-time video feed from anappropriately-mounted camera (or cameras) directed along the user'seyeline; hence the data and problem domain are fundamentallytwo-dimensional. VST AR provides absolute control over the finalappearance of visual stimulus, and facilitates registration andsynchronization of captured video with any synthetic augmentations. Verywide fields-of-view (FOV) approximating natural human limits are alsoachievable at low cost. However, VST gear tends to be bulky and incuradditional latencies associated with image capture.

OST AR eyewear, on the other hand, has a direct optical path allowinglight from the scene to form a natural image on the retina. This naturalimage is essentially the same one that would be formed without ARglasses, possibly with some loss of brightness due to attenuation by thecombining optics. A camera is used to capture the scene for automatedanalysis, but its image does not need to be shown to the user. Instead,computed annotations or drawings from an internal display aresuperimposed onto the natural retinal image by (e.g.) direct laserprojection or a half-silvered mirror for optical combining. In atraditional OST AR application, the majority of the display typicallyremains blank (i.e., black) to avoid contributing any photons to thefinal retinal image; displayed augmentations produce sufficient light tobe visible against this background. The horizontal field-of-view overwhich annotations can be projected tends to be limited to a central 25to 50 degrees, but there is no delay between real-world events and theirperception. Furthermore, the scene image has no artifacts due toimage-sensor sampling, capture, or processing. However, synchronizingaugmentations becomes more challenging and user-dependent calibrationmay be needed to ensure proper their registration. Finally, OSTpossesses an inherent degree of safety that VST lacks: if the OSThardware fails, the user can still see the environment.

Augmented Reality and Low Vision. The primary task of visual-assistanceeyewear for low-vision sufferers does not match the most common usemodel for AR (whether VST or OST), which involves superimposingannotations or drawings on a background image that is otherwise faithfulto the reality seen by the unaided eye. Instead, assistive devices needto dramatically change how the environment is displayed in order tocompensate defects in the user's vision. Processing may include contrastenhancement and color mapping, but invariably incorporates increasedmagnification to counteract deficient visual acuity. Existing devicesfor low-vision are magnification-centric, and hence operate in the VSTregime with VST hardware. Some alternative methods employ an OST-basedAR platform, but install opaque lens covers that completely block allenvironmental light from entering the retina—since a camera supplies theonly visible image via an internal display, it is exclusively a VSTsystem.

Hybrid See-Through. This methodology describes an AR platform that isnominally OST for its development effort, but employs a unique combinedVST/OST methodology (hybrid see-through, or HST) to produce its finalretinal image. Doing so permits the best characteristics of eachtechnique to be effectively exploited while simultaneously avoiding orameliorating undesirable aspects. Specifically:

The wide field of view associated with VST can be maintained for theuser in spite of the narrow active display area of the OST-basedglasses;

Absolute control over the final retinal image details is achieved (as inVST) for the highest-acuity central area covered by the internaldisplay;

A fail-safe vision path exists at all times (as in OST), regardless ofthe content of the internal display—and whether or not that display isfunctioning;

A recently-identified need specific to low-vision is addressed andremedied.

There are three aspects to HST implementation that together engender itseffectiveness: spatial partitioning, tailored image processing, andelimination of focus ambiguity.

Spatial Partitioning. There are typically three types of viewing in aHST AR system, corresponding to three characteristically distinct pathsfor OST AR light rays as they travel from a viewed scene into an eye andonto its retina. Only two types are fundamentally different, but it isconvenient for the purposes of this document to distinguish the third.

Consider the drawings in FIGS. 3A-3B, which depict front and side viewsof one eye within a set of notional OST AR glasses employing ahalf-silvered mirror to combine incoming scene light with the imageshown on an internally-mounted display. In this example, the display ismounted in the top frame of the glasses and points downward such thatthe mirror directs its reflected image into the eye; the same mirrorallows light from the environment to pass directly through it into theeye. Other mounting orientations and optical combining strategies exist,but this one adequately illustrates the three relevant types of lightpaths for all of them.

In both drawings, labels A, B, & C represent light rays originating inan environmental scene, directed toward the eye and travelling throughthe pupil and onto the retina. Label A indicates light that travelsdirectly from the scene to the retina without intersecting anynon-trivial lenses or mirrors. Labels B and C denote light that travelsfrom the scene and into the retina, but only after passing through thecombining mirror.

The difference between the two is that C intersects the region of themirror where the internal display also projects its output. Light fromthis display does not interact with scene light at the combiner, sothere is no intrinsic difference between types B and C other than thissimple fact of geometry. However, the importance of the distinction isclarified immediately below:

Type A. For portions of the field-of-view that are not within range ofthe internal display (and also not completely or partially blocked byhalf-silvered mirrors or other optics, a direct and natural light pathfrom the scene to the retina exists. This OST path cannot activelyparticipate in AR since the display cannot affect its retinal image, butits existence preserves the user's existing peripheral vision andmaintains a fail-safe degree of visual capability regardless of what theinternal display is showing.

Type B. For portions of the field-of-view that intersect the combiningmirror but do not overlap the projected internal display, there may besome loss of brightness due to attenuation as light rays of type B passthrough the combining optics. Type B rays are otherwise identical tolight of type A, and can provide significant OST peripheral vision aboveand below the internal display image.

Type C. In a traditional AR application these light rays, whichintersect the projection of the internal display onto the mirror, wouldbe blended on the retina with the image presented on the display. InHST, however, this combining process—which is the very essence of OSTAR—is deliberately prevented by blocking type C light so that thecentral visual field comprises only content originating in the display.Thus a defining paradigm is subverted, and OST eyewear locally takes oncharacteristics of VST.

It is important to note that blocking type C rays is not an obviouschoice to make. OST AR displays are typically capable of providing lightpower sufficient to overwhelm the direct scene image on retina, causingthe brain to perceive only the dominant image. The additional utilitygranted by blocking type C light will be described in a later section.

It is the partitioning of angular space into explicit OST and VSTregions that lends HST its name. The remaining two aspects serve toamplify its utility.

Tailored Image Processing. In HST ASR, the image provided by theinternal OST display replaces the natural retinal image that wouldnormally be produced by type C light rays. Like VST AR display content,it is derived in real-time from an eyewear-mounted camera video streamwith additional processing applied. However, OST displays have a muchnarrower field of view than their VST cousins, so more sophisticatedcomputation is needed to provide utility.

The specific processing used with HST is described earlier and notconsidered in detail here. Relevant features for the present discussionare:

The internal display contributes a dense replacement image for theentire central visual field, not merely an overlay of sparse ARannotations;

Image processing is user- and task-specific, but almost invariablycontains some amount of magnification over at least a portion of itsextent (implying that a traditional OST-style overlay would not beviable);

The final displayed image is adjusted to appear to blend smoothly intothe peripheral areas of vision (formed from light rays of type A and B)where the active display does not extend.

Tailoring the central visual field to suit the user and current taskleverages a hallmark capability of the VST paradigm—absolute controlover the finest details of the retinal image—to provide flexiblecustomization and utility where it is most needed. Whereas traditionalOST AR produces displayed images that neatly coexist and integrate withthe natural scene that they overlay, low-vision and HST AR must applycarefully-selected and painstakingly-tuned nonlinear distortions tosatisfy their users. Even though the underlying platform isfundamentally OST, careful blending restores a naturally widefield-of-view for a seamless user experience despite the narrow activedisplay region.

Elimination of Focus Ambiguity. For sections of the field-of-view thatcoincide with the projected internal display (i.e., the same sectionsviewing the replacement image), the direct optical light path from thescene to the retina is blocked in HST. This can be accomplished byoccluding the scene-facing portion of the half-silvered mirror in theoptical combiner. (Analogous procedures for blocking this light will beobvious in other configurations.) It is important to note that only theportion of the combiner having an image from the internal displayprojected onto it (gray shading in FIGS. 3A-3B) should be blocked,because the surrounding region can contribute significant peripheralvision (particularly at the top, between the upper edge of theinternally-generated image and the upper frame of the eyewear).

The rationale behind this non-obvious modification to the standard OSTAR configuration is developed immediately below.

Recall that traditional AR operations in the OST regime allow light fromthe scene to travel directly to the retina and form a natural imagethere; the internal display can then be used to overpower this naturalimage so that augmentations are visible to the user. In this low-visionapplication, it is desirable to overpower the entire scene (within theactive display limits) with an enhanced (and generally magnified)replacement.

Typical OST AR hardware is easily capable of producing a bright enoughimage to overwhelm the natural scene image under practical lightingconditions. For users with normal vision, this is a perfectly reasonableoperating mode, and HST as described above is viable without blockingtype C light from reaching the retina. For some low-vision users,unfortunately, this does not hold true.

To understand why, consider the task of reading a book while using OSTglasses without any additional magnification and without blocking anydirect light path. Normal reading distance without AR gear is 16-24inches and requires accommodation of the lens within the eye to focus alegible image on the retina. The output from the internal display on ARglasses is typically collimated to appear to be originating from adistance of 8-10 feet, allowing the eyes to relax and avoid eyestrain.Without blocking the direct light path, there will be two superimposedimages formed on the OST AR user's retina—the natural image focused inthe near field, and the display image focused in the far field.

Users with normal vision can readily select between the two nearlyidentical images, shifting focus at will. In test sessions, however,low-vision users exhibited poorer reading ability even when cameraimages clearly exhibited increased contrast: they were not able todetect and exploit the contrast cues that normally-sighted individualsuse to drive their focus response to completion, and hence were not ableto focus successfully on either competing image.

Blocking the direct path that coincides with the internal displayalleviates this problem.

What is claimed is:
 1. A visual aid eyeglass device, comprising: acamera configured to capture images of a scene; an integrated processorconfigured to modify the images to produce corrected images of thescene; one or more displays configured to present the corrected imagesof the scene to one or more eyes of a low-vision user; one or moreopaque lens covers disposed on the one or more displays and configuredto prevent external light from passing through the one or more displays;wherein when the visual aid device is worn by the low-vision user, theone or more opaque lens covers are positioned and configured to block anexternal central light path that coincides with the one or more displaysso that a central visual field of the low-vision user comprises only thecorrected images of the scene on the one or more displays, and whereinthe one or more opaque lens covers are also positioned and configured tosimultaneously allow an external peripheral light path that is adjacentto the one or more displays and is not blocked by the one or more opaquelens covers to reach the eye of the low-vision user to preserve thelow-vision user's existing peripheral vision of the scene.
 2. The visualaid eyeglass device of claim 1, further comprising a frame, wherein thecamera and the one or more displays are mounted on the frame.
 3. Thevisual aid eyeglass device of claim 2, wherein when the visual aiddevice is worn by the low-vision user, the external peripheral lightpath passes between an edge of the one or more displays and the frame ofthe eyeglass system and is not blocked by the one or more opaque lenscovers.
 4. The visual aid eyeglass device of claim 1, wherein theintegrated processor is configured to modify a magnification of theimages.
 5. The visual aid eyeglass device of claim 1, wherein theintegrated processor is configured to modify an amount of magnificationover at least a portion of the images.
 6. The visual aid eyeglass deviceof claim 1, wherein the integrated processor is configured to modify acontrast of the images.
 7. The visual aid eyeglass device of claim 1,wherein the integrated processor is configured to adjust the correctedimages to blend smoothly from the central visual field of the low-visionuser into the peripheral vision of the low-vision user.
 8. The visualaid eyeglass device of claim 7, wherein the integrated processor isconfigured to modify a magnification of the images.
 9. The visual aideyeglass device of claim 7, wherein the integrated processor isconfigured to modify an amount of magnification over at least a portionof the images.
 10. The visual aid eyeglass device of claim 7, whereinthe integrated processor is configured to modify a contrast of theimages.
 11. The visual aid eyeglass device of claim 1, wherein theintegrated processor is configured to apply color mapping to the images.12. The visual aid eyeglass device of claim 7, wherein the integratedprocessor is configured to apply color mapping to the images.